Three-dimensional imaging system for robot vision

ABSTRACT

The present invention provides a three-dimensional imaging system for robot vision, which is capable of three-dimensional positioning of objects, object identification, searching and tracking of an object of interest, and compensating the aberration of the system. The three-dimensional imaging system for robot vision comprises one or more camera systems, each of which has at least one variable focal length micromirror array lens, an imaging unit, and an image processing unit. The variable focal length micromirror array lens used in the three-dimensional imaging system for robot vision has unique features including a variable focal length, a variable focal axis, and a variable field of view with a fast response time.

REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part of, and claims priority toU.S. patent application Ser. No. 10/806,299 filed Mar. 22, 2004 (DocketNo. 1802.03), U.S. patent application Ser. No. 10/822,414 filed Apr. 12,2004 (Docket No. 1802.04), U.S. patent application Ser. No. 10/983,353filed Nov. 8, 2004 (Docket No. 1802.013Con), U.S. patent applicationSer. No. 10/872,241 filed Jun. 18, 2004 (Docket No. 1802.011), U.S.patent application Ser. No. 10/893,039 filed Jul. 16, 2004 (Docket No.1802.012), U.S. patent application Ser. No. 10/979,619 filed Nov. 2,2004 (Docket No. 1802.18), and U.S. patent application Ser. No.10/979,624 filed Nov. 2, 2004 (Docket No. 1802.19), all of which arehereby incorporated by reference.

FIELD OF INVENTION

The present invention relates to a three-dimensional imaging system forrobot vision comprising at least one micromirror array lens.

BACKGROUND OF INVENTION

Robots have been widely used in the industry to replace human who had toperform dangerous and repetitive tasks. While earlier robots are usuallyrequired to manipulate objects in a limited range with less or noflexibility like robot arms in assembly lines, recent robots have becomemore intelligent and perceptive. Such robots are capable of performingmore complex and difficult tasks including navigation, inspection,self-learning, and self-calibration thanks to advanced technologies forcomputation resources and sensory systems with lower cost. Someapplications of these advanced robots include, but not limited to,housekeeping, underwater/space exploration, surgery with precision, andmine finding and mining.

Many aspects of sensory systems being used in robotics are adopted fromhuman biological systems. Human senses including sight, hearing, smell,touch, and taste are very acute and efficient considering small sizesand fast processing times of their sensory systems. In the early stagesof adopting these human sensing procedures, it was very difficult tocreate corresponding artificial sensory systems because of complexity,limited resources, and a lack of knowledge. Since then, a lot of effortshave been put into researches for these areas, and huge progresses havebeen made. Especially, human vision systems are relatively wellunderstood and adopted in most advanced robot systems as a primarysensory system.

Some important features of human vision include three-dimensionalperception, continuous tracking of a moving object, rapid objectidentification, and the like. Among them, three-dimensional perceptionis a fundamental element since it allows other features available. Manyadvanced robot systems that are required to perform navigation and/ormanipulation in known or unknown environments have adoptedthree-dimensional vision systems, which collect and processenvironmental information surrounding a robot, and let the robotproperly respond to stimuli without interruption from outside sources.

Typically, three-dimensional vision for robots can be accomplished byusing stereo vision or optical flow methods, in which two images arecompared in order to determine the three-dimensional location of anobject. The former uses images taken by two parallel cameras that aredisposed to view the object from different angles at the same time asdisclosed in U.S. Pat. No. 5,432,712 to Chan, while the latter usesimages taken at two different times by a single camera as disclosed inU.S. Pat. No. 5,109,425 to Lawton. Both methods require to findcorresponding points of two different images using certain criteria suchas color, shape, contrast, or other representative features. However,these processes can be very erroneous and time-consuming.

Research suggests that the human vision system is more efficient andeffective in that it is capable of a rapid eye movement to the point ofinterest and contrasting high central visual resolution with lowperipheral visual resolution in a wide field of view as disclosed inU.S. Pat. No. 5,103,306 to Weiman et al. These aspects demand fastchanges of the optical axis and field of view of a lens system. However,it is difficult to accomplish such efficiency and effectiveness in aconventional robot vision system since those changes are usuallyperformed by a complicated macroscopic servo mechanism.

To overcome the drawbacks of existing technologies, a desirable robotvision system requires a high-speed, accurate, miniaturized, andinexpensive three-dimensional imaging system.

SUMMARY OF INVENTION

The present invention provides a three-dimensional imaging system forrobot vision, which is capable of three-dimensional positioning of anobject, object identification, searching and tracking the object, andcompensation for the aberration of the system.

One aspect of the invention is to provide a three-dimensional imagingsystem for robot vision that generates an all-in-focus image andthree-dimensional position information of an object.

The three-dimensional imaging system for robot vision comprises at leastone camera system having a lens system including at least one variablefocal length micromirror array lens (MMAL), an imaging unit, and animage processing unit.

The variable focal length MMAL comprises a plurality of micromirrors.The following US patents and applications describe the MMAL: U.S. Pat.No. 6,934,072 to Kim, U.S. Pat. No. 6,934,073 to Kim, U.S. patentapplication Ser. No. 10/855,554 filed May 27, 2004, U.S. patentapplication Ser. No. 10/855,715 filed May 27, 2004, U.S. patentapplication Ser. No. 10/857,714 filed May 28, 2004, U.S. patentapplication Ser. No. 10/857,280 filed May 28, 2004, U.S. patentapplication Ser. No. 10/893,039 filed May 28, 2004, and U.S. patentapplication Ser. No. 10/983,353 filed Mar. 4, 2005, all of which arehereby incorporated by reference.

The variable focal length MMAL is suitable for the three-dimensionalimaging system of the present invention since it has a fast focusingspeed and a large range of focal length, and since it can be made tohave either a small or a large aperture just adding more micromirrors tocover the aperture area.

The imaging unit includes one or more two-dimensional image sensorstaking two-dimensional images at different focal planes. The detail forthree-dimensional imaging using the variable focal length MMAL can befound in U.S. patent application Ser. No. 10/822,414 filed Apr. 12,2004, U.S. patent application Ser. No. 10/979,624 filed Nov. 2, 2004,and U.S. patent application Ser. No. 11/208,115 filed Aug. 19, 2005.

The image sensor takes two-dimensional images of an object or scene withone or more focal planes that are shifted by changing the focal lengthof the variable focal length MMAL. The image processing unit extractssubstantially in-focus pixels or areas from each two-dimensional imageto generate a corresponding in-focus depthwise image. Each in-focusdepthwise image represents a portion of the object or scene having thesame image depth. Based on the known focal length of the two-dimensionalimage, the known distance from the lens to the image plane, and themagnification of the lens, three-dimensional position information of aportion of the object corresponding to each pixel of the in-focusdepthwise image can be obtained. The focal length of the variable focallength MMAL can progressively increase or decrease, or vary in aselected order within a focal length variation range of the variablefocal length MMAL such that any portion of the object or scene is imagedsubstantially in-focus at least once. A set of in-focus depthwise imagestaken at different focal lengths with a fast imaging rate represents theobject or scene at a given moment. The object can remain still or bemoving. For the case that the object is moving, the movement of theobject can be ignored when the imaging rate is fast enough. The numberof in-focus depthwise images representing the object at a given moment(number of depths) depends on the depth resolution requirement and thefocusing speed of the variable focal length MMAL, and may increase for abetter image quality. There are several methods for the image processingunit to generate an all-in-focus image of the object or scene fromin-focus depthwise images thereof. Recent advances in both the imagesensor and the image processing unit make them as fast as they arerequired to be. Three-dimensional position information of a portion ofthe object corresponding to each pixel of the all-in-focus image isobtained in the same way as in the in-focus depthwise image case. Allthe processes to obtain an all-in-focus image and three-dimensionalposition information of the object are achieved within a unit time whichis at least persistent rate of the human eye.

Three-dimensional position information of an object or scene is anecessary element for robot's object manipulation and navigation. Also,the all-in-focus image and three-dimensional position information of theobject enables the object identification to be more accurate and robustsince three-dimensional object identification is less subject to theobject orientation, illumination, and other environmental factors. Thedetail for the three-dimensional imaging system for pattern recognitionusing the variable focal length MMAL can be found in US patentapplication Ser. No. (11/294,944) filed Dec. 6, 2005.

The next aspect of the invention is to provide an optical tracking unitusing a three-dimensional imaging system for robot vision that performssearching and tracking an object of interest. Similar to theaforementioned human vision system, it is desirable that the object issearched in a wide field of view with low resolution images while beingidentified and tracked in a narrow field of view with higher resolutionimages. The variable focal length MMAL of the present invention has alarge range of focal length variation, which can offer a variable fieldof view (a variable magnification); a wider field of view with a largemagnification and a narrow field of view with a small magnification. Thefield of view is changed without macroscopic movements of the lenssystem because each micromirror of the variable focal length MMAL isadjusted for varying the focal length and actuated by the electrostaticforce and/or electromagnetic force.

Tracking systems usually require that the object of interest be in thecenter of an image sensor. However, this entails a camera attitudecontrol or a robot body attitude control for large-scale tracking. Inthe optical tracking unit of the present invention, the optical axis ofthe variable focal length MMAL can be adjusted in a limited range bycontrolling each micromirror of the variable focal length MMALindependently without using macroscopic servo mechanisms, which allowsthe robot vision system to be simplified and lightly weighted whencompared to that having conventional tracking systems. The focusingspeed of the variable focal length MMAL is so fast that a moving objectcan be identified and tracked. The principles of maintaining focus on afast moving object are described in detail in U.S. patent Ser. No.10/896,146 filed Jul. 21, 2005.

Another aspect of the invention is to provide a three-dimensionalimaging system for robot vision that can compensate the aberration bycontrolling each micromirror independently. The presentthree-dimensional imaging system can produce a substantially sharp imagethrough entire area without blurring or vignetting.

Still another aspect of the invention is to provide a small and compactthree-dimensional imaging system for robot vision which can be used in amicro-robot vision system having limitation in space. Unlikeconventional stereo vision systems that require at least two camerasystems, the present invention can determine three-dimensional positioninformation of an object or scene using only a single camera system, andthis renders a simpler camera calibration and a more compact imagingdevice. Further, since the variable focal length MMAL can be made tohave a small aperture, and the magnification and optical axis can beadjusted without macroscopic movements of the lens system, thethree-dimensional imaging system for robot vision of the presentinvention can be made small and compact.

The present invention of the three-dimensional imaging system for robotvision using the variable focal length MMAL has the followingadvantages: (1) the system provides all-in-focus images; (2) the systemprovides three-dimensional position information of an object or sceneusing a single camera system; (3) The imaging processes to obtain anall-in-focus image and three-dimensional position information of anobject or scene is achieved faster than the persistent rate of the humaneye; (4) the system has a large variation of field of view since thesystem has a large range of focal depth; (5) the system uses a largefield of view for searching and a small field of view for identifying ortracking an object of interest; (6) the system has a variable opticalaxis to locate an object image in the center of an image sensor; (7) thesystem can identify and track a moving object; (8) the system has a highdepth resolution; (9) the production cost of the system is inexpensivebecause the variable focal length MMAL is inexpensive; (10) the systemis very simple because there is no macroscopic mechanical displacementor deformation of the lens system; (11) the system is compact andsuitable for the small as well as large robot vision system; (12) thesystem demands low power consumption when the variable focal length MMALis actuated by electrostatic force.

Although the present invention is briefly summarized herein, the fullunderstanding of the invention can be obtained by the followingdrawings, detailed description, and appended claims.

DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic illustration of a three-dimensional imaging systemfor robot vision;

FIG. 2 is a schematic diagram showing how in-focus depthwise images areobtained from two-dimensional images with different focal planes;

FIG. 3 is a schematic illustration of a three-dimensional imaging systemfor robot vision with a variable optical axis;

FIG. 4 is a schematic illustration of a three-dimensional imaging systemfor robot vision with a variable optical axis and a variable field ofview;

FIG. 5 is a schematic representation for optical axis changes in theMMAL;

FIG. 6A is a schematic diagram showing how a refractive Fresnel lensreplaces an ordinary single-bodied lens;

FIG. 6B is a schematic diagram showing how a reflective MMAL replaces anordinary single-bodied mirror;

FIG. 7A is a schematic plan view showing a variable focal length MMALthat is made of many micromirrors;

FIG. 7B is an enlarged detail plan view of the micromirrors;

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

FIG. 1 illustrates a robot vision system 11 of a robot 12 working invarious environments with different tasks including manipulation, objectidentification, searching and tracking an object of interest, andnavigation in known and unknown territories. The image sensor (notshown) receives two-dimensional images with different focal planes thatare shifted by changing the focal length of the variable focal lengthMMAL. The image processing unit (not shown) generates all-in-focusimages and calculates three-dimensional position data of objects 13.Using all-in-focus images and three-dimensional position information ofthe object, the robot vision system can perform manipulation, objectidentification, searching and tracking a still or a moving object,navigation, and the like. It is efficient that the imaging system uses awide field of view 14 to search an object of interest from the scene andthen use a narrow field of view 15 to identify and track the objectprecisely. When using the narrow field of view, the object being trackedmay be lost more easily (increase in dropout rate). Because the focallength of the MMAL can be quickly changed, a time-sharing technique canbe utilized in order to use a narrow field of view for high resolutionimages and a wide field of view for low tracking dropout. It is alsopossible to adjust the optical axis of the MMAL to keep a moving object16 in the center of the image sensor for a limited range without usingmacroscopic servo mechanisms.

FIG. 2 shows how a MMAL 21 takes two-dimensional images 22A, 22B, 22Cwith the focal planes 23A, 23B, 23C. The MMAL 21 comprises a pluralityof micromirrors 24. Each micromirror 24 is controlled to change thefocal length of the variable focal length MMAL 21. The focal length ofthe variable focal length MMAL 21 is changed by rotation and translationof each micromirror 24, which are controlled by electrostatic and/orelectromagnetic force. Two-dimensional images 22A, 22B, 22C are takenwith the depth information which corresponds to the position of thefocal plane. The two-dimensional image 22A has in-focus image LI at thefocal plane 23A, which is the image of a portion L of an object 25.Images MD, ND of portions M, N of an object 25 are defocused. The imageprocessing unit determines the in-focus pixels LI from thetwo-dimensional images 22A. The two-dimensional image 22A with depthinformation gives in-focus pixels LI corresponding to the focal plane23A. The two-dimensional images 22B, 22C with the second and third focalplane 23B, 23C are processed in the same manner as the first focal plane23A to get in-focus images with depth information. Also the imagingsystem can perform the automatic focusing function for finding theobject distance by changing the focal plane of the MMAL.

FIG. 3 schematically illustrates a three-dimensional imaging system forrobot vision with a variable optical axis according to the oneembodiment of the present invention. The three-dimensional imagingsystem for robot vision 31 comprises a lens system (32, 33), an imagingunit 34, and an image processing unit 35. The lens system includes anobjective lens 32, and a variable focal length MMAL 33, opticallycoupled to the objective lens 32, configured to change the focal planeby changing the focal length of the MMAL 33. The imaging unit 34receives two-dimensional images of an object 36 with different focalplanes that are shifted by changing the focal length of the variablefocal length MMAL 33. The image depth of the focal plane is obtainedfrom the focal length of the variable focal length MMAL 33. The imageprocessing unit 35 extracts substantially in-focus pixels or areas fromoriginal two-dimensional images taken at different focal planes togenerate in-focus depthwise images and provides three-dimensionalposition information of the object 36. A set of in-focus depthwiseimages taken at different focal lengths with a fast imaging raterepresents the object at a given moment. Then, the image processing unit35 generates an all-in-focus image and three-dimensional positioninformation of the object. By controlling individual micromirrors of thevariable focal length MMAL 33, the optical axis can be adjusted, as willbe explained in FIG. 5. Since the focal plane and the optical axis ofthe MMAL can be changed without macroscopic movements, the arrangementof optical elements in FIG. 3 requires a small space and allows acompact size three-dimensional imaging system for robot vision.

FIG. 4 illustrates a three-dimensional imaging system for robot visionwith a variable optical axis and a variable field of view (a variablemagnification) according to the other embodiment of the presentinvention. The three-dimensional imaging system for robot vision 41comprises a lens system, an imaging unit 42, and an image processingunit 43. The lens system comprises an objective lens 44, and a variablefocal length MMAL 45, optically coupled to the objective lens 44,configured to change the focal plane by changing the focal length of theMMAL 45. The lens system also comprises an auxiliary lens 46 or group oflenses to change the field of view and image resolution. Further, thelens system comprises one or more auxiliary lenses for increasing thenumerical aperture of the imaging system.

The lens system can comprise the second variable focal length MMAL 47for a variable magnification. The first and second variable focal lengthMMAL 45, 47 are optically coupled and controlled to change themagnification of the system wherein the image of an object is opticallymagnified and to change the focal plane to form two-dimensional imagesin-focus at a given magnification. The objective lens 44 and theauxiliary lens 46 provide additional magnification. The field of view isadjusted without macroscopic movements of the lens system or time delaysince each micromirror 48 of the variable focal length MMALs 45 and 47is adjusted and actuated by electrostatic and/or electromagnetic force.

The image processing unit 43 generates an all-in-focus image andthree-dimensional position information of an object 49 usingtwo-dimensional images received from the imaging unit 42. The variablefocal length MMAL 45 and 47 change their focal lengths so fast that theimaging processes to obtain the all-in-focus image and three-dimensionalposition formation of the object are achieved faster than thepersistence rate of the human eye. Further, by controlling individualmicromirrors of variable focal length MMALs, the optical axis of thelens system can be adjusted, as will be explained in FIG. 5. Since theMMAL can be made to have a small aperture, and the field of view andoptical axis can be adjusted without macroscopic movements of the lenssystem, the arrangement of optical elements in FIG. 4 requires a smallspace and allows a compact size three-dimensional imaging system forrobot vision.

The FIG. 5 shows how the optical axis of the MMAL changes. A bunch oflight is focused by the MMAL 51. In FIG. 5A, a cube object 52 is imagedonto the image plane. The light 53A from the object 52 is reflected byeach of the micromirror 54. The reflected light 55A is focused onto thefocal plane 56A of the image and finally makes an image of a cube 57A inthe image sensor. During the focusing process the optical axis isdefined as a surface normal direction 58A of MMAL 51.

As shown in FIG. 5B, the MMAL can make a different image 57B from adifferent object 59 without macroscopic movements. By changing therespective angles of the micromirrors 54, the MMAL focuses the light 53Bfrom the sphere 59 onto the image focal plane 56B. The reflected light55B is focused onto a focal plane 56B and makes the image of the sphere57B. This time the optical axis is changed by an angle and becomes thesurface normal direction 58B of a micromirror.

FIG. 6A schematically shows how a refractive Fresnel lens 61A replacesan ordinary single-bodied lens 62. FIG. 6B shows how a MMAL 61Breplaces, replacing an ordinary single-bodied mirror 63. The MMALincludes a plurality of micromirrors 64, and each micromirror 64 iscontrolled to form a MMAL 61B and to change the focal length of thelens.

In order to obtain a bright and sharp image, the variable focal lengthMMAL must meet the two conditions for forming a lens. One is that allthe rays should be converged into the focus, and the other is that thephase of the converged rays must be the same. Even though the rays havedifferent optical path lengths, the same phase condition can besatisfied by adjusting the optical path length difference to be integermultiples of the wavelength of the light. Each facet converges rays toone point, and rays refracted or reflected by different facets have anoptical path length difference of integer multiples of the incidentlight.

To change the focal length of the MMAL, the translational motion and/orthe rotational motion of each of the micromirrors are controlled tochange the direction of light and to satisfy the phase condition of thelight.

The variable focal length MMAL is also an adaptive optical componentcompensating the aberration of the imaging system by controlling thetranslational motion and/or the rotational motion of each micromirror.

FIGS. 7A and 7B show that the micromirrors 71 are arranged to form manyconcentric circles. The micromirrors 71 are arranged in a flat plane asshown in FIG. 6B.

The response speed of the micromirror 71 can exceed the persistent rateof the human eyes times the number of depths unless the depth resolutionrequirement is extremely high. It is possible to make the focal lengthchange within hundreds of micro-seconds. The range of numerical aperturechange of the MMAL is large since the range of focal length variation ofthe MMAL is large. So, the MMAL can have a greater range of imagedepths, which is an essential requirement for a three-dimensionalimaging system.

1. A three-dimensional imaging system for robot vision comprising: (a) alens system, comprising a variable focal length micromirror array lens(MMAL), configured to change the focal plane by changing the focallength of the variable focal length MMAL; (b) an imaging unit, opticallycoupled to the lens system, configured to receive an object image fromthe lens system and to sense the object image; and (c) an imageprocessing unit, communicatively coupled to the imaging unit, configuredto process the object images sensed by the imaging unit and to generatean all-in-focus image and three-dimensional position information of theobject.
 2. The three-dimensional imaging system for robot vision ofclaim 1, wherein the imaging unit comprises at least one two-dimensionalimage sensor taking the two-dimensional images at different focalplanes, wherein the focal plane is changed by change of focal length ofthe variable focal length MMAL.
 3. The three-dimensional imaging systemfor robot vision of claim 1, wherein the lens system further comprisesan auxiliary lens or group of lenses to change the field of view of thelens system.
 4. The three-dimensional imaging system for robot vision ofclaim 1, wherein the lens system further comprises one or more auxiliarylenses to increase the numerical aperture of the lens system.
 5. Thethree-dimensional imaging system for robot vision of claim 1, whereinthe lens system further comprises an auxiliary lens or group of lensesto change the image resolution of the three-dimensional imaging system.6. The three-dimensional imaging system for robot vision of claim 1,wherein the lens system with the variable focal length MMAL furthercomprises another variable focal length MMAL for a variable field ofview.
 7. The three-dimensional imaging system for robot vision of claim1, wherein the variable focal length MMAL comprises a plurality ofmicromirrors.
 8. The three-dimensional imaging system for robot visionof claim 7, wherein each micromirror is controlled to change the focallength of the variable focal length MMAL.
 9. The three-dimensionalimaging system for robot vision of claim 7, wherein each micromirror iscontrolled to change the optical axis of the lens system.
 10. Thethree-dimensional imaging system for robot vision of claim 1, whereinthe imaging system can perform the automatic focusing function.
 11. Thethree-dimensional imaging system for robot vision of claim 1, whereinthe variable focal length MMAL compensates the aberration of the system.12. The three-dimensional imaging system for robot vision of claim 1,further comprising an object identification unit using all-in-focusimages received from the image processing unit.
 13. Thethree-dimensional imaging system for robot vision of claim 12, whereinthe object identification unit uses three-dimensional positioninformation of the object received from the image processing unit. 14.The three-dimensional imaging device for robot vision of claim 1,wherein the imaging processes to obtain an all-in-focus image andthree-dimensional position information of the object is achieved fasterthan the persistent rate of the human eye.
 15. The three-dimensionalimaging system for robot vision of claim 1, further comprising anoptical tracking unit, coupled to the lens system, configured to changethe optical axis of the lens system by changing the optical axis of thevariable focal length MMAL to track an object of interest.
 16. Thethree-dimensional imaging system for robot vision of claim 15, whereinthe optical tracking unit is coupled to the lens system, configured tochange the field of view of the lens system by changing the focal lengthof the variable focal length MMALs.
 17. The three-dimensional imagingsystem for robot vision of claim 15, wherein the optical tracking unitincludes a time sharing algorithm to use a wide field of view forsearching and a narrow field of view for identifying and tracking theobject of interest.
 18. The three-dimensional imaging system for robotvision of claim 15, wherein the optical tracking unit is coupled to thelens system, configured to measure the distance to the object ofinterest.
 19. The three-dimensional imaging system for robot vision ofclaim 15, wherein the optical tracking unit is coupled to the lenssystem, configured to measure the size of the object of interest.